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ABSTRACT 

Automated schemes are needed to 
classify multi-spectral remotely 
sensed data. Human intelligence is 
often required to correctly interpret 
images from satellites and aircraft. 
Humans succeed because they use 
various types of cues about a scene 
to accurately define the contents of 
the image. Consequently, it follows 
that computer techniques that 
integrate and use different types of 
information would perform better than 
single source approaches. 

This research illustrated that 
multispectral signatures and 
topographical information could be 
used in concert. Significantly, this 
dual source tactic classified a 
remotely sensed image better than the 
multispectral classification alone. 
These classifications were 
accomplished by fusing spectral 
signatures with topographical 
information using neural network 
technology. 

A neural network was trained to 
classify Landsat multi-spectral 
images of the Black Hills. Bands 4, 
5, 6 and 7 were used to generate four 
classifications based on the spectral 
signatures. A file of georeferenced 
ground truth classifications were 
used as the training criterion. The 


network was trained to classify 
urban, agriculture, range and forest 
with 65.7% correct. Another neural 
network was programmed and trained to 
fuse these multispectral signature 
results with a file of georeferenced 
altitude data. This topological file 
contained 10 levels of elevations. 
When this non-spectral elevation 
information was fused with the 
spectral signatures the 
classifications were improved to 
73.7% and 75.7%. 


INTRODUCTION 

Automated schemes are needed to 
classify multi-spectral remotely 
sensed data. For example, the 
upcoming Earth Observing System (EOS) 
will generate massive quantities of 
data that must be managed quickly 
(Dorfman, 1991; Short, 1991). Access 
to the resulting data and information 
should be quick and user friendly. 
Campbell and Cromp (1990) call for a 
user friendly system that is based on 
user domain-specific knowledge and 
goals. This concept requires that the 
data system be based on object- 
oriented storage and retrieval 
procedures that incorporate 
information about the image (Dorfman, 
1991). Fekete has recommended a 
sphere quadtree technique for 
subdividing and relating spherical 
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data into a data base. This technique 
relies on the identification of image 
contents such as coast lines. 

If these recommended data base- 
information systems are to be based 
on content and knowledge about the 
data, then real time classif ication 
algorithms will be required. Data 
storage techniques, such as the 
sphere quadtree (Fekekte) or object 
oriented information, are based on 
the contents of the image/data. Data 
storage will depend on access codes, 
indexes, or keys specific to the 
content of data. These codes and 
indexes would be determined as the 
data arrives and prior to storage 
into the data base. An accurate and 
automated classification technique 
would be the basis for determining 
these indexes that will be used for 
cataloging and filing data. Due to 
the large quantity of data coming 
from the EOS, this classification- 
indexing and storage process should 
occur in real time or near real time 
to avoid building a backlog. Not only 
will it be necessary to transmit and 
store EOS data efficiently, but also 
EOS data should be categorized 
somehow during the transmit or 
storage process. 

While EOS data management will 
be important, rapid or near real time 
mult ispectral remotely sensed data 
classification is important in its 
own right. There are potential 
satellite image applications that 
depend on rapid access to 
classification results. Images should 
be classified without the delay 
associated with most processing 
techniques. The results would be 
transmitted to the user in a timely 
fashion. This rapid classification 
and delivery would support the 
feasibility of many new applications. 
For example, fishermen could respond 
quickly to recent current shifts. 
Short term illegal wild cat mining or 
deforestation could be identified and 
arrested. Natural disasters such as 


oil spills could be monitored as they 
progress . 

Real time classification 
techniques do not exist; however, 
neural network technology promises to 
allow us to automatically classify 
images in real time. The technique is 
simple yet can be deployed with 
parallel neural processing integrated 
circuits. These processors are 
relatively cheap and available for 
multispectral analysis (Harston, 
Zhant & Stewart, 1991; Kagel, 1991). 

The neural network approach has 
classified various remotely sensed 
multispectral images (Campbell, Hill 
& Cromp, 1989; Benediktsson, Swain & 
Ersoy, 1990; Cromp, 1991; Harston & 
Schumacher, 1991; Kulkarni, 1990; 
Eberlein & Yates, 1991; Decatur, 
1989). Decatur's work was with the 
synthetic aperture radar (SAR) HH, 
HV, and W components of the return 
at the L band (1.225 GHz) and the 
others were with visual and infrared 
bands. Some of their results can be 
seen in Table III. While the results 
compare well with statistical 
classification techniques, better 
performance is desirable. It was 
hypothesized that fusion of spectral 
signatures with additional 
information might improve 
performance . 

Human intelligence is often 
required to correctly interpret 
images from satellites and aircraft. 
Humans succeed because they use 
various types of cues about a scene 
to accurately define the contents of 
the image. Consequently, it follows 
that computer techniques that 
integrate and use different types of 
information would perform better than 
single source approaches. Work to 
date in our laboratory supports this 
supposition (Harston, 1991 a, b & c). 

This research illustrated that 
multispectral signatures and 
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topographical information could be 
used in concert. Significantly, this 
dual source tactic classified a 
remotely sensed image better than the 
multispectral information alone. 
These classifications were 
accomplished by fusing spectral 
signatures with topographical 
information using neural network 
technology. 


METHOD 

The data came from a Landsat 
Multispectral Scanner (MSS) image of 
the Black Hills. Thematic mapper (TM) 
Bpectral bands 4, 5, 6 and 7 were 
represented as intensity values from 
0 to 255 in 512 by 512 byte image 
files. Additionally, files of 
elevation and ground truth data were 
available. The ground truth showed 
that broad contiguous areas were 
assigned to single classifications. 
There were 22 potential 
classifications, which covered urban, 
farm, range, forest, and water as 
major groupings. The four classes of 
data used in this study were urban, 
farm, range, and forest. These and 
the other data files were 
georef erenced . 

The type of neural network used 
was the three layer feedforward 
networks with one layer as the 
hidden layer. The delta rule was used 
to train the output layer and 
backpropagation was used to train the 
hidden layer. All work was done on 
MS-DOS 386/486 VGA microcomputers and 
the code was written in C. 

One neural network was trained 
to classify Landsat multi-spectral 
images of the Black Hills. Bands 4, 
5, 6, and 7 were used to generate 
four classifications based on the 
spectral signatures. These classes 
included or collapsed several of the 
classifications found in the ground 
truth into urban, farm, range, and 
forest categories. The file of 


condensed georef erenced ground truth 
classifications was used as the 
training criterion. This network was 
called the spectral signature 
network. 

Another neural network was 
trained with the four bands of TM 
data and a topography file of 
altitude or elevation data. This 
topological file contained 10 levels 
of elevations. These images/files 
were georef erenced to each other, and 
the ground truth file was used for 
training. This network was referred 
to as the fusion network. 

Training for both networks 
consisted of hand picked samples from 
the larger image. The experiment was 
conducted twice, resulting in two 
spectral signature networks and two 
fusion networks. The second set of 
networks was tested with additional 
samples taken from the same image. 
These samples were taken from 
intersection points of a grid taken 
at 50 pixel (horizontal) and 25 pixel 
(vertical) locations on the upper 
part of the image. There were 2 5 test 
points taken from urban, farm, range, 
and forest areas that resulted in 
only one range and one urban testing 
sample. This kind of grid sampling 
and random sampling may be roughly 
representative of the types of data 
in the image but does not obtain 
equal numbers of cases for each 
category. 


RESULTS 

Both spectral signature 
networks learned 65.7% of the 
training sets (60,021 training trials 
for the second network) . The first 
fusion network learned 73.7% of the 
training set, and the second fusion 
network learned 75.8% at 63,035 
training trails. Further training 
resulted in decreased levels of 
performance. 
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The second set of networks , 
both spectral and fusion, were tested 
with other data taken from the 
multispectral image. The spectral 
signature network generalized to 
these novel data points at 52%, and 
the fusion network correctly 
classified 60% of these test cases. 

These results (see Table I for 
results) were carefully reviewed with 
the image in view. Sixteen percent of 
the errors appeared to be correctly 
classified. That is, the ground truth 
did not appear to be correct from the 
visual examination of the image. 
Additionally, 4% of the errors were 
not clear from the visual image, and 
the ground truth classification could 
be debated. The results improved 16% 
for both the signature and fusion 
network test results when the scores 
were corrected for the obvious (not 
the 4% border line) ground truth 
errors. 

Regardless of the corrections, 
it is clear that the fusion of 
altitude information with spectral 
signatures improved the learning. 
This improvement was 8% in the first 
set of networks and 10% in the second 
set of networks. Even the testing 
results improved by 8% with the 
second set of networks. 

A detailed analysis of the 
errors indicated that the greatest 
number of errors came from 
misclassif ied farm data. Keep in mind 
that there were more test samples 
from farm areas than from other 
areas. The performance in each class 
can be seen in Table II. There was 
only one range test sample and that 
one was misclassif ied. This resulted 
in a 100% error rate for the range 
class. 

The fusion with elevation data 
improved the farm scores from 60% to 
80%. Unfortunately the forest 
performance was decreased from 87.5% 
to 75%. The test sample size was 


small at 25 cases so interpretation 
of the results may be limited. The 
misclassif ied range sample was one of 
the debatable or border line cases. 
These results are also found in Table 
II. 


CONCLUSIONS 

The use of altitude data with 
the spectral signatures improved the 
performance. This fusion of image and 
topographic data was simple to do 
with the neural networks. The 
elevation data improved the farm land 
classification but degraded the 
forest classification to a lesser 
extent. However, the results were 
positive overall and suggest that 
classification performance could be 
further improved if other types of 
data were included in the neural 
classification process. 

The initial impression that the 
learning and test results were low 
should be interpreted in relationship 
to the results from similar 
classifications. For example, as seen 
in Table III, other types of 
statistical classification are also 
low (Benediktsson, Swain & Ersoy, 
1990; Duda & Hart, 1973). In general, 
the neural techniques performed 
better, except with the multisource 
technique that used information in 
addition to the spectral data 
(Benediktsson, Swain & Ersoy, 1990). 
This multisource statistical 
technique included Landsat MSS, 
elevation, slope, and aspect data. 
This additional data improved the 
classification technique to 61%. 
Clearly, this result argues for the 
fusion with, or inclusion of, 
additional cues, regardless of the 
classification technique used. 

Classification of raw spectral 
data without any clean up is also 
poor as seen in Table I with 55% 
(Campbell, Hill & Cromp, 1989) and 
52% or 60% in this study. Neural 
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studies often correct or select the 
data in some way. Campbell, Hill, & 
Cromp ( 1989 ) used only non-boundary 
pixels for training. Homogeneous 
fields were developed for training 
and testing by Benediktsson, Swain, 
& Ersoy, 1990 . In the present case, 
the classification was corrected by 
visual inspection of the test cases. 
Some of these selection or correction 
procedures resulted in respectable 
test scored at 70 % (Campbell, Hill & 
Cromp, 1990 ) and 76 % in this study. 

Given the improved 
classifications by fusing non 
spectral data with the spectral 
signature, possibly other 
supplementary information can be used 
by the neural network system to 
improve performance. A shadow file 
might be used to improve the 
classification of forests on both 
sides of the mountain. In another 
study, pixel patterns based on 
brightness and texture were 
classified within each MSS TM band 
(Harston, 1 9 9 1 d ) . These 
classifications were fused with an 
additional neural network that 
resulted in improved performance. 
Possibly, the texture, signature, 
altitude, and other data can be fused 
with neural technology to obtain even 
higher test performance. 

The potential for classifying 
incoming Earth Observing System (EOS) 
data in real time is genuine. See 
papers authored by Short, Campbell, 
Fekete, Dor f man, and Cromp at the 
GSFC for a description of the 
importance of this problem. As 
indicated in our multi-spectral work 
to date, meaningful results are 
possible; however, higher levels of 
performance may be possible. It seems 
reasonable that more can be done with 
multi-spectral data when additional 
non-spectral information is 
integrated with the spectral results. 
Further work with seasonal, urban, 
hazy, and cloudy images is needed. 
Ultimately, a system that could 


classify regardless of variations or 
conditions could categorize incoming 
data in real time. Such 
categorizations would be useful for 
the Intelligent Data Management ( I DM ) 
project as a basis for defining, 
cataloging, and referencing images 
for a data base. 
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TABLE I 


NEURAL NETWORK TRAINING 
AND TESTING RESULTS 



EXPERIMENT I . 
SPECTRAL NN 
PUSION NN 


65 . 7X 
7 3 • 7 X 



EXPER IMENT I I . 

SPECTRAL NN 60,021 65. 7X 52X 68% 

PUSION NN 63,035 75. ©X 60X 76X 


TABLE II 

CORRECTED PERFORMANCE FOR EACH CLASS 



CORRECTED NEURAL NETWORK CLASSIFICATION 

GROUNO = — ■ ' ' -■ " 1 ===== — 

TRUTH URBAN FARM RANGE FOREST 

URBAN 1 OOX ... 

PARM 20X BOX 

RANGE 1 OOX OX 

FOREST - 12.5X 12.5X 75X 
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TABLE III 


MULT I SPECTRAL IMAGE 
CLASSIFICATION PERFORMANCES 


NEURAL NETWORKS 
TRAINING TESTING 

CAMPBELL, HILL 


& CROMP , 1989 

ANY 

PIXEL 

48 % 

55 % 

WASH I NOTON, DC 
W/ GROUND TRUTH 

NON- 

BOUNDRY 

PIXELS 

66 % 

70 % 


BENEDIKT S SON , 

c \jJ a T KJ » P R C H Y 


STATIST ICAL • 

CLASSIFICATION 

9 TT A x n ft c n a w T y 

1990 


ED 

ML 

MD 

MULTI- 

SOURCE 

COLORADO MOUNTAINS HOMOOENEOUS 

HELDS 

95 % 

52 . 5 % 47 % 

49 % 

50 % 

61 % 


CROMP , 1991 

BLACK HILLS LANDSAT HSS 60 % 


HARSTON , 1 99 1 

MLC W / NN 

100% 

61 % 

MURFREESBORO LANDSAT MSS 
MULT I SPECTRAL FUSION SYSTEM 

1 SAMPLE 
/CLASS SET 

100% 



BRIGHTNESS 

85 % 

43 % 


BRIGHTNESS 
A TEXTURE 

75 % 



HARSTON & 

SPECTRAL 



SCHUMACHER , 
1 99 1 

SIONATURE 
W/ ROAD 
DETECTOR 

95 % 

63 % 

TIMS IMAGES 

NN SYSTEM 




HARSTON & 
SCHUMACHER 9 
199 1 

W/ GROUND 
TRUTH 


CORRECTED 
FOR VISUAL 
INTERPRETATION 

BLACK MILLS LANDSAT MSS 

SPECTRAL 

SIONATURE 

65 . 7 % 

52 % 

68 % 

SPECTRAL 

SIONATURE 

W/ALTITUDE 

75 . 8 % 

60 % 

76 % 


« ED: MINIMUM EUCLIDEAN DISTANCE 
MLt MAXIMUM LIKELIHOOD METHOO 
MO: MINIMUM MAHALANO0IS DISTANCE 
MULT I SOURCE: STATISTICAL MULTISOURCE ANALYSIS 
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